Summary

This adds an ML module called logs_ui_analysis, which contains a job definition to detect anomalies in the log rate.

⚠️ Some fields are designed to be overridden at setup time using #42946:

• in the job log-entry-rate
  • analysis_config.bucket_span
  • data_description.time_field
• in the datafeed datafeed-log-entry-rate:
  • aggregations.buckets.date_histogram.field
  • aggregations.buckets.date_histogram.fixed_interval
  • aggregations.buckets.aggregations[timestampField]
  • aggregations.buckets.aggregations[timestampField].max.field
  • aggregations.buckets.aggregations.doc_count_per_minute.bucket_script.script.params.bucket_span_in_ms

closes #42593

Implementation notes

• The datafeed scales the doc_count to one minute using a bucket_script aggregation called doc_count_per_minute. That aggregation is used as the summary_count_field_name in the job configuration. The scaling has several advantages:
  • Denominators of 1 in the unit reduce the risk of misinterpretation (doc_count / minute compared to doc_count / (15 minutes).
  • Future changes of bucket span sizes are simpler because the result's unit is not as tightly coupled to it.
• The max timestamp aggregation is not included in the datafeed, because its name depends on the time field name. Since the additive nature of the overrides makes a removal of the predefined aggregation impossible, it needs to be specified solely as an override at setup time.

Testing hints

The module can be deployed via a call to Kibana's ML module setup api at /api/ml/modules/setup/logs_ui_analysis. The POST requests body needs to specify a few parameters, e.g.:

{
  "indexPatternName": "filebeat-*",
  "prefix": "kibana-logs-ui-testspace-default-",
  "startDatafeed": true,
  "jobOverrides": [
    {
      "job_id": "log-entry-rate",
      "analysis_config": {
        "bucket_span": "900000ms"
      },
      "data_description": {
        "time_field": "@timestamp"
      }
    }
  ],
  "datafeedOverrides": [
    {
      "job_id": "log-entry-rate",
      "aggregations": {
        "buckets": {
          "date_histogram": {
            "field": "@timestamp",
            "fixed_interval": "900000ms"
          },
          "aggregations": {
            "@timestamp": {
              "max": {
                "field": "@timestamp"
              }
            },
            "doc_count_per_minute": {
              "bucket_script": {
                "script": {
                  "params": {
                    "bucket_span_in_ms": 900000
                  }
                }
              }
            }
          }
        }
      }
    }
  ]
}